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In large multipurpose surveys, it is common to select the 
sample systematically proportional to some measure of size (PPS) that is 
correlated with an important variable of interest . Assuming the frame is 
sorted in a useful deterministic manner, systematic sample methodologies 
provide an additional control on the sample allocation, beyond the control 
provided from the stratification. This makes it less likely to select a "bad 
sample." This should reduce the variability of the estimates as compared to a 
comparable nonsystematic selection procedure. The problem with systematic 
samples is that variance estimators are biased. This paper presents a 
bootstrap variance estimator, which can have less bias than standard 
methodologies, such as half-sample replication. The results will be 
demonstrated with a simulation study based on an important National Center 
for Education Statistics survey, the Schools and Staffing Survey. An appendix 
demonstrates the consistency of the proposed estimators. (Contains 8 tables 
and 12 references.) (Author/SLD) 
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Abstract 



In large multipurpose surveys, it is common to select the sample systematically proportional to some measure of size 
(PPS) which is correlated with an important variable of interest. Assuming the frame is sorted in a useful 
deterministic manner, systematic sample methodologies provide an additional control on the sample allocation, 
beyond the control provided from the stratification. This makes it less likely to select a ‘bad sample’. This should 
reduce the variability of the estimates as compared to a comparable nonsystematic selection procedure. The problem 
with systematic samples is that variance estimators are biased. This paper presents a bootstrap variance estimator, 
which can have less bias than standard methodologies, such as half-sample replication. The results will be 
demonstrated with a simulation study based on an important National Center for Education Statistics’ survey-The 
Schools and Staffing Survey. 

Key Words: Simulation, Half-Sample Replication 

The author would like to acknowledge Michael Cohen for his helpful comments on the draft manuscript. 
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1.0 Introduction 

Systematic probability proportionate to size sampling (PPS) is a common selection method for establishment surveys. 
One way of selecting a PPS sample, given a frame and measure of size, is to do the following within each sampling 

stratum ( h ): Partition the Primary Sampling Units (PSUs) on the frame into n h groups, so that the sum of the measures 

of size in each group is equal. For the sum of the measures of size to be equal within each group, some PSUs may be 
split into two adjacent groups, with some positive measure of size in each group. The sampling interval is the total 
measure of size in each group. Within each group, the first PSU is assigned a cumulative measure of size equal to the 
PSU’s measure of size. The second PSU’s cumulative measure of size is the sum of the previous PSU’s (first PSU) 
cumulative size plus the current PSU’s (second PSU) measure of size. This process continues until each PSU in a group 
is assigned a cumulative measure of size. The cumulative measure of size for the last PSU in each group is equal to the 
sampling interval. A random distance ( d h ) is chosen between 0 and the length of the sampling interval. The first PSU 

with a cumulative measure of size larger or equal to d h , within the first group, is the first PSU selected in the sample. 
The rest of the sample is selected by making the same cumulative measure of size comparison for each group. A total of 
n h PSUs are selected within each stratum. (See Wolter 1985, pp. 283-286 for more details). 

The measure of size for some PSUs may be larger than the sampling interval. There are two ways of handling this. All 
such PSUs can be excluded from the selection process and place in sample with certainty. The stratum sample sizes can 
be adjusted and new sampling intervals computed. The alternative is to select the sample without modification and 
adjust the weighting or second stage sample sizes to accommodate PSUs selected multiple times. The former is 
generally considered more efficient because the number of distinct selected PSUs will equal the original sample size, 
while the latter will be somewhat less. 

Systematic sampling procedures are efficient in terms of ease of selection and lowering sampling error. For this reason 
they are used extensively in large-scale surveys. Since each stratum systematic sample is selected using a single random 
start ( d h ), the sample can be viewed as a sample of size one, where each sample consists of a single sample cluster of 
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n h PSUs. Therefore, it is impossible to produce an unbiased variance estimator. However, a number of biased 
methodologies are used for variance estimation. 

These methodologies generally take one of two forms: 1) assume the systematic sample can be approximated by a 
simpler sample design with a known variance estimator or 2) assume the response variable follows some super- 
population model and a variance estimator is produced appropriate for that model. Both these approaches allow for 
grouping of PSUs, so variances can be computed within groups. Wolter (1985, chapter 7) provides a good discussion of 
a number of systematic sample variance estimators that can be classified into one of these two forms. An example, using 
balanced half-sample replication (BHR) is provided below. 

BHR is a widely used variance replication methodology for complex survey designs. It is designed for samples where 
two primary sampling units (PSUs) within each stratum are selected with replacement. With BHR, choosing one PSU 
within each stratum generates a half-sample. A number of half-samples are generated by alternating which PSU, within 
stratum, go into the half-samples. The BHR variance is the simple variance of the half-sample estimates. Through a 
balancing process of the half-samples, the BHR variance estimate, for linear estimates, equals the direct sample variance 
estimate. 

BHR can be adapted to designs where more than two PSUs are selected in a stratum by consecutively pairing selected 
PSUs, after placing them in the original order of selection; and assuming each pair is a stratum for variance estimation 
(variance stratum). If without-replacement sampling is used then a finite population adjustment can be applied. See 
Wolter (1985, pp. 1 10-152) for a more complete description of BHR. 

In order to use BHR with systematic PPS sampling, it must be assumed that a PPS selection can be approximated by the 
deep stratification induced by the pairing described above. This assumption is reasonable, considering that the first sort 
variable, ignoring the lack of independence between breaks in the variable, can be considered an implicit stratification. 
However, BHR also assumes that the variance estimate is proportional to the inverse of the sample size. In section 2.0, it 
will be demonstrated, through a simulation study, that systematic sampling variances are not necessarily inversely 
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proportional to the sample size. Two possible reasons for this are: 1) there is a finite population adjustment effect and 2) 
the variance decreases by some function other than 1 / n h . From the simulation study, the amount of deviation from the 

1 / n h model is a function of the response variable, as well as, the sample size. Hence, it is unlikely that the error can be 
corrected using a simple finite population correction. Therefore, BHR is not expected to perform well in this situation. 

One methodology that does not necessarily assume the variance is inversely proportional to the sample size is the 
bootstrap (Efron, 1982). Given an estimate 0(X, , X 2 X n ) , with X x , X n i.i.d. and distributed according 

to a distribution function F(X x ) , the bootstrap variance estimate for ©is based on the observation that 

V(0) = <T 2 (0, rt,F)( i.e., the variance of 0 is a function of 0 , the sample size n , and F ). Given 0 and n , 

V (0) = G (F ) . By analogy, the bootstrap variance, V* (0) , equals G ( F ) , where F is the empirical probability 

b 

distribution of X . If <7 2 () is unknown then 1^(0) can be estimated byl/Z^G'-O*) 2 , for 

/'=! 

0*,/ = ltoZ?. ©* is computed like 0 is computed, except X,* , X* 2 X* is used instead of X x , X n , 

with the X*' s being independent realizations from F(x). This process is repeated B times to obtain B 0* *s. 
X x , X X* n is called a bootstrap sample with a bootstrap sample size ( n * ) n . 

The bootstrap procedure can be applied to stratified simple random sampling by applying the above bootstrap procedure 

__ L 

within each sampling stratum. In this case, V*(^0 equals 1/ n h [(n h —1)/ n h ] being the usual stratum 

h = 1 

population variance. Comparing this to the usual variance estimator, this methodology has two problems: 1) the finite 
population adjustment is missing and 2) there is a scaling bias term [(n h — 1) / n h ] . If the sample rates are high then the 

missing finite population correction adjustment can be significant. If n h is small, which is quite common in finite 
population sampling, then [( n h — 1) / n h ] can be large. With the basic BHR sample design, where n h equals 2, the bias 
is -50%. Many adjustments have been suggested to correct these deficiencies. 
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In the BHR type sample design, where n h equals 2 for all h and the estimate of interest is linear, setting rc* = 1 
corrects for the [( n h —1 )/n h ] bias. Since BHR type designs are with-replacement designs, there is no finite 

population adjustment. The bootstrap variance estimator, therefore, becomes unbiased and consistent (see Efron 1982). 
This bootstrap variance estimator is similar to the BHR variance estimator. The difference is that PSU’s selected for the 
BHR replicates are specified to produce the exact sample variance estimate, while PSUs are randomly selected for the 
bootstrap replicates and may not equal the exact sample variance. 

When n h is greater than 2, other more complicated adjustments have been proposed. The simplest of these is to, adjust 
n* h to eliminate the finite population correction and [( n h — 1 )/ n h ] bias problems (see McCarthy and Snowden 1985). 
However, n* h is not unique. Rao and Wu (1988) propose a rescaling bootstrap, which determines n* h to match the third 

order moments between 0 — £0 and 0 * — £ 0 * after rescaling V* (©* ) to be unbiased for any n* h . Sitter (1992a) 
proposed a bootstrap designed for the case where PSUs are selected without replacement. In this bootstrap, 
X,* , X 2 ,.**» X *. are generated from a series of without replacement samples from X, , X 2 X„ . By selecting 

X,*, X 2 ,...,X*. without replacement, the original sample design is followed more closely. All these methodologies 

ft h 

directly select X,* , X 2 X ** from the original X, , X 2 X n and adjust n* h to correct for any biases and 
inconsistencies introduced in the bootstrap process. 

To further mimic the actual sample design (see Gross 1980; Chao and Lo 1985), a bootstrap-frame can be generated to 
select the bootstrap samples. If N h = k h n h (N b being number of PSUs in the frame for stratum h ), the bootstrap- 

frame is generated by replicating each of the n h sampled PSUs k h times. Bootstrap samples are then selected using the 

original sampling methodology (i.e., simple random sampling without replacement). The bootstrap variance estimates 
based on these bootstrap samples no longer have a finite population correction bias, but have a scaling bias term 

k h (n h ” l)(see ShaoandTu 1995, p. 250). By adjusting k h and n* h according to Sitter (1992b), this bias 
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All the bootstrap procedures described above assume simple random sampling (with or without replacement) within 
each stratum. Sitter (1992b) proposed a bootstrap variance estimator for the Rao-Hartley-Cochran sampling scheme 
(Cochran 1977, pp. 266-267). This is a TPS type’ sampling scheme because it is similar to a PPS systematic selection 
where the stratum frame is placed in a random order before sample selection. The Rao-Hartley-Cochran sampling 
scheme independently selects one PSU, within each of n h groups, proportional to some measure of size. The groups are 

generated by randomly assigning a specified number of PSUs into each of the n h groups. Since the sum of the measures 
of size within a group are not equal the procedure is not strictly PPS either. The bootstrap methodology is similar to the 
bootstrap- frame procedures described above. First, a bootstrap frame is generated. Next, n* h is chosen to eliminate any 

biases. The bootstrap-PSUs are then randomly placed into one of ri h groups. Next, one bootstrap-PSU is independently 

selected within each of the n* h groups, generating a bootstrap sample. Before selecting each bootstrap sample, the 

bootstrap frame is re-randomized. A number of bootstrap samples are generated, as well as the appropriate bootstrap 
estimate. The Monte Carlo variance estimator of the bootstrap estimates is an estimate of the bootstrap variance 
estimator. 

The bootstrap procedures described above assume the variance is inversely proportional to ri h (i.e., n h and /or k h are 
chosen knowing exactly how the true variance is related to n h ). If the true variance has a known relationship to n h , 

different than proportional to 1 !n h , then n* h , most likely, can be adjusted to compensated for the biases in the 
bootstrap variance. If the relationship is unknown, as is the case with systematic sampling (PPS or equal probability), 
then a simulation study can be done to compute an n* h that is approximately unbiased. This is the approach used in this 
paper to produce a bootstrap variance estimate for stratified PPS systematic sampling. A bootstrap-frame will be used to 
reduce any bias due to sampling without replacement. n* } will be computed by means of a simulation study, comparing 

the bootstrap variance for a specified n* h with an estimate of the true variance, to reduce any additional biases. A super- 
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population model will be introduced to determine how to randomize the bootstrap-frame before selecting the bootstrap 
samples. Given the super-population model, the bootstrap variance estimator will be shown to be consistent. 
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2.0 Using the BHR Model with Systematic Sampling 

The BHR model assumes systematic sampling can be approximated by a deep stratification introduced by pairing 
consecutive sampling PSUs. For this to work, the stratum variances must be proportional to 1 / n h , since BHR makes 

this assumption. (When all BHR assumptions are true, this follows from V BHR (X) — V((X ] +X 2 )/2) — 
\/2V (X ] ) , where subscript 1 and 2 represents the estimate based on the first and second PSUs respectively selected 
in each stratum.) If this assumption is not true then the BHR model is unlikely to produce accurate results. There are two 
reasons for systematic sampling to violate this assumption. The first reason deals with any implicit finite population 
correction in the variance to reflect sampling without replacement. If the sampling rates are high, this could be a 
significant contributor to the violation of the 1 / n h assumption. The second reason is the correlation between PSUs 

within a systematic sample. As n h increase or decreases, these correlations may change dramatically because of the 

original sort ordering. Unlike the finite population adjustment, this effect can be noticeable even when the sampling rates 
are small. 

To investigate the 1 / n h assumption, a simulation study is done, using the National Center for Education Statistics 
(NCES) elementary/secondary private school frame. Four thousand systematic samples are selected with sample sizes of 
p i n h , where p. = 1, 0.75, 0.5 and 0.25 . By computing the simple variance of the 4,000 simulation estimate, an 
estimate of the true variance is computed. This is done for estimates of total students, teachers and schools. If the 
variance is proportional to 1 / n h , then the ratio V (X t )p k / V (X / )p . - 1 should be close to 0; where / represents 
the estimate type (total students, teachers or schools) and k , j represents the sample size ( n h , .75 n h , .5 n h or .25 n h ). 
When the ratio is less than 0, the systematic sample variance decreases faster than the n h assumption would imply; 
and when it is greater than 0, the systematic sample variance decreases slower than the 1 / n h assumption would imply. 
A negative ratio means that BHR should overestimate the variance, while a positive ratio means that BHR should 
underestimate the variance. This relationship is not necessarily true, since V(X { ) includes an unknown implicit finite 
population correction whose impact on the variance as the sample changes is unknown. 
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The results in table 1 demonstrate that sometimes the ratio is close to 0. Other times, it is a great deal different than 0. 
The systematic sampling variance does not necessarily decrease faster than the 1 1 n h assumption would imply; 

sometimes its decrease is slower. This is an indication that BHR will not necessarily produce an overestimate of the 
variance, which is a common assumption among sampling statisticians. When there is a large difference from 0, the 
magnitude is dependent on the variable. This seems to imply, since the sampling rates are not high, (especially with the 
.51.25 comparison), that the violation of the 1 1 n h assumption is due to the initial sort ordering (i.e., the within sample 
correlation). 

It should be noted that the table 1 results exaggerate the true impact of the 1 / n h assumption. Using the 

1/ n h assumption, the ratio, used in the table, adjusts the variance with the smaller sample size to approximate the 

variance with the larger sample size. This approximation uses the smaller sample estimate’s unknown finite population 
correction. Since the true finite population correction is likely larger than the one used in the approximation, the absolute 
value of the true impact of the 1 / n h assumption should be expected to be smaller than what table 1 indicates. 

The important conclusion from this example is that variance estimates, based on designs using systematic sampling, will 
not necessarily be proportional to 1 / n h , as n h increases or decreases. When this occurs, an important BHR assumption 

is violated, and the BHR variance estimator should not be expected to perform well when the magnitude of the violation 
is large. 

The statements concerning the proportionality of the variance estimate are qualified with ‘as n h increases or decreases’. 
The importance of this qualification can be seen with equal probability systematic sampling. Here, the variance can be 
expressed proportional to l/^(e.g., V(y h ) = [(N h - n h )/ N h ][Sl sf In J[1 + (n h - l)p wsf ] , see (Cochran 

1977, p. 209). If S* st and p wsr are constant for an arbitrary n h then V(y k ) would be approximately proportional to 
n h , as n h increases or decreases. However, both S^, sr and p wst are within systematic sample population estimates. 
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This implies that as n h changes, the systematic samples change; hence S~ s( and p wst also change by some unknown 
function of n h . Therefore, even though V(y h ) is proportional to 1 / n h for fixed n h , as n h increases or decreases, the 
variance may not be proportional or even closely proportional to 1 / n h . In section 3.5, a similar result will be presented 
for unequal probability systematic sampling. 



In terms of BHR where V BHR (X) = l/2V(X l ) , theS^and fi wst used in V(A r ,)may be different than the 
S^ >sf and p from V(X) . So, V BHK (X) may over or under estimate the true variance (V(X) ) based on the 
relationship between the two sets of S* st and p wst . 

Table 1. -Measurement of degree the true systematic sampling variance is proportional to 1 1 n h with respect 
to different sample sizes 1 



Stratum 

(h) 


n „ /N „ 

(%> 


Number of Teachers 


Number of Students 


Number of Schools 


R100/50 2 

(%) 


R50/25 2 

(%) 


R75/25 2 

(%) 


R100/50 2 

<%) 


R50/25 2 

(%) 


R75/25 2 

(%) 


R100/50 2 

(%) 


R50/25 2 

(%) 


R75/25 2 

(%) 


01911 


2.0 


-31.2 


19.5 


55.2 


14.1 


-13.5 


0.3 


-28.0 


-8.5 


100.8 


01912 


2.8 


-27.0 


24.5 


8.2 


-2.5 


6.6 


6.0 


-14.3 


12.5 


22.3 


01913 


6.1 


-14.3 


-1 1.9 


3.8 


20.9 


19.2 


-3.6 


-3.0 


3.2 


10.2 


01914 


3.4 


-23.8 


-26.6 


2.5 


-5.3 


8.8 


2.8 


-19.3 


-3.7 


-7.4 


01921 


19.6 


23.5 


25.3 


18.0 


56.5 


141.9 


110.1 


19.6 


34.7 


10.8 


01922 


25.7 


-32.7 


14.9 


132.1 


-34.0 


18.5 


15.2 


-16.4 


35.4 


27.1 


01923 


13.7 


-4.3 


-51.7 


-31.2 


6.0 


12.0 


175.5 


-8.4 


10.1 


-10.1 


01924 


12.2 


-46.3 


38.2 


-45.2 


-37.6 


-26.5 


- 7.9 


-37.6 


28.9 


-13.9 


01931 


4.5 


23.7 


-7.5 


-21.3 


18.0 


-4.5 


-17.7 


4.6 


-6.3 


-11.0 


01932 


4.9 


2.4 


1.1 


-29.6 


-25.8 


4.0 


-37.5 


3.9 


-9.4 


-0.9 


01933 


6.2 


18.5 


43.5 


51.7 


-21.4 


-24.0 


-52.0 


14.4 


47.9 


100.6 


01934 


4.3 


-20.4 


-9.6 


-34.4 


-26.3 


-16.9 


-18.9 


-7.0 


1.5 


-3.8 



1 Negative numbers represent how much more efficient the variances are than the 1 / n h assumption. 



Positive numbers represent how much less efficient the variances are than the 1 1 n h assumption. 
2 R a lb is the comparison for a % of original sample size to b % of original sample size. 
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3.0 Bootstrap Variance Model 

To address the situation when the systematic variance in not proportional to 1/ n h , a bootstrap variance estimator is 
proposed in this paper, which is less dependent on the 1 / n h assumption than the BHR estimator. This section first 

describes the necessary super-population model; next, a consistency theorem for the bootstrap estimator is presented; by 
example, the super-population model, used in the proposed bootstrap procedure, is demonstrated; next, the mechanics of 
the bootstrap procedure is presented; and finally, the consistency of the bootstrap procedure is established. We begin by 
describing the super-population model. 

3.1 The General Super-Population Model 

c„ 

Let s ih , the i ,h possible sample from stratum h , be a systematic PPS sample such that s ih = [^J s ihc , where the 

C=1 

S ihc ’s are a partitioning of the sample S ih and C h is the number of partitions within stratum h . 

Each sampled PSU j E S ih is assigned a known weight ( w . ) based on the selection probability: 



Wj=T h /(n h e hj )., 



where T h = ^e h 



H 



e h j is the measure of size for PSU j in stratum h , 



n h is the number of sampled PSUs in stratum h , 



( 1 ) 

( 2 ) 



and N h is the number of PSUs in stratum h . 
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Let p h =n h ln, (3) 

where n = n h . 

h 

We will assume that the p h ’s are constant as n increases. 

For a random variable X ; evaluated for each PSU j , assume the joint distribution of Y } — (T h /(p h e j))X } 

«l/.r 

given s ihc is n F hc (Yj), where F hc (Yj ) is a distribution function, and n [hc is the number of sampled PSUs in 
;=i 

partition C within stratum h . 

Assume the distribution of Y* c is independent of Y* c , C ^ C , given s ih , where Y* c = (y ,,..., Y )' on s ihc 
and Y, k '=(y„..,y, i ,)'on S „,. 

Q 

The joint distribution of Y^ = (^jY/ lc , given S ih , is J**J ( F hc ) n,,lc . 

ces ih c = i 

The distribution of Y /; given s ih can be summarized in the following way: Within a partition of S ih the Y j ’s are 
i.i.d. The Yj ' s E S jhc and the Y ■ ’s E S ihc * , C ^ C , are independent, but not identically distributed. 

Theorem 1 

Given the above assumptions and definitions, the bootstrap variance estimator of X = is W hJ Xj given the 

h je h 

* ^ 

super-population model is consistent, as n o o , provided F hc (y) F hc (y) and fl yhc jl yhc , as n — > °° . 

The proof is provided in the appendix. 
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This result can be generalized by noting that implementing the bootstrap methodology does not require knowledge of 
the conditional distributions, F h c . The only thing required is the knowledge that the PSUs within s ich are i.i.d. and 

PSUs between s ich and S ic > h , C ^ C , are independent. This implies PSUs within s ich must be randomized together 
as a group. 

The following theorem states this generalization. 

Theorem 2 

The required assumptions are: 

c ih 

1) a systematic sample ( s ih ) has a known partition (i.e., S ih = s ihc ); 

C=1 

2) X = II ¥r 1/n ii y j is the estimate of interest; 

. h jeh h jeh 

3) as n increases the sample allocation between stratum remains constant (i.e., the p h ’s are constant as n 
increases); 

4) for PSUs in S ihc , the y . ’s are conditionally i.i.d. given S ihc and are generated from an otherwise unspecified 
distribution function F hc (y)G 3 2j - 3 2j being defined in the appendix and E(y } ) = JU Yhc ; 

and 

5) Y* c is conditionally independent of Y* c , C ^ C , given S ih . 

It then follows that the bootstrap variance estimator of X given s ih generated from the bootstrap estimates 
x;=unjy y* , where the y* ’s are generated from F hc (y) , is consistent, as n —> oo ? provided 

h je h 

* * * 

F hc (y) —> F hc (y) and fX yhc /J yhc , as fl — > 00 . JU yhc is the bootstrap expectation of y within a partition. 
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3.2 Bootstrap Model Example 

c h 

In practice, the statistician never knows the required partitioning ( S ih = |^J S ihc ). However, the statistician usually 

orders the frame before sample selection. With this ordering, the statistician is implicitly assuming that nearby PSUs 
are similar, at least in terms of the most important response variables. This implicit assumption can be used to 
develop a partitioning that approximately meets the required assumptions. This approach is similar to the BHR 
approach, so that the starting point of the bootstrap and the BHR are the same. Any differences in results reflect the 
divergence of assumptions at this point. 

An example is provided below. However, it is first necessary to redefine the meaning of the term sampling interval. 
In the introduction, the sampling interval was the total measure of size within a sampling group. From now on, a 
sampling interval will refer to the PSUs within a sampling group. Consecutive sampling intervals are consecutive 
sampling groups. 

Example 

For a fixed even numbered sample size ( n h ), the elements of the partition ( S ich ) can be determined by consecutively 
pairing sampling intervals, after the frame has been placed in its original sort ordering. All samples have the same 
partitioning (i.e., the partitioning is only a function of stratum, — S ch , c = 1 to C h ) and each S ich ( S ch ) has exactly 

two PSUs. The marginal distribution of Y* r is (F hc ) 2 • If y is distributed as n(jU c , O c ) with distribution 

functionO^ 0 (y)then the distribution of Y* r is a (}>))• In terms of consistency, it is assumed that the 

partitioning remains fixed as the sample size increases and more PSUs are selected within a partition. 

This “type” of partitioning is used in the bootstrap procedures proposed in this paper. Implementing the partitioning 
in the bootstrap processes is similar to the BHR model, in that, for both methodologies, all the replicate variability is 
introduced through the variability within consecutive pairs of selected PSUs. The BHR replicates reflect variability 
by alternating PSUs within pairs in and out of the replicates, while the bootstrap replicates reflect variability by 
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randomizing bootstrap-PSUs generated from consecutively paired PSUs (see section 3.3 for a description of the 
bootstrap-PSU process). The bias in the bootstrap procedure gets smaller as the number of PSUs per partition ( s ch ) 
gets larger (see theorem 2). Since the proposed partitioning has only two PSUs per partition, the bias may be large; 
hence, setting the bootstrap sample size, n* h , equal to n h should underestimates the bootstrap variance (see the 

bootstrap discussion in the introduction). Using n* h = n h / 2 , as in BHR, may have less bias. The bootstrap, unlike 
BHR, does not have to use n* h = n h / 2 . Any n* h between 1 and n h can be used and may have less bias than either 
nj 2 or n h . In this sense, the bootstrap model is more flexible than the BHR model. 

The actual bootstrap sample size must be computed through a series of trial and error simulations, comparing and 
estimate of the true variance with the bootstrap variance for a specific bootstrap sample size. The bootstrap sample 
size that minimizes the bias in the bootstrap variance is used in the final implementation. The trial and error process 
is necessary because there is no direct formula that expresses the systematic variance as a function of n h , as n h 

increase or decreases, such as being proportional to 1 1 n h . 

Determining n* h through a simulation provides a robust variance estimate because V* (X h ), by construction, will 
be almost unbiased, even if the model assumptions are false. The disadvantage of the simulation is that it can only be 
implemented with frame variables. However, if n* h is relatively flat for non-frame variables, the bootstrap replicate 
weights should be applicable for those variables, too. 



O 

ERIC 



An additional observation about this partitioning is: 

If the partitioning methodology described above correctly models the distribution of X ; the n h ’s are even and 
increase by multiples of C h then the E ( X . |j h 9 S ) = . K , a constant; where E refers to the expectation 
with respect to the super-population model. Therefore, 

V(X ; ) = EV(X -\s h ' s) + V E(X { : ^ ' s) = EV(X . ’s) , where 1 refers to the selection of the S h 9 S . In this 
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situation, since the bootstrap variance estimator is consistent for V (X ; Js,, ’s) , the bootstrap variance is consistent to 
an estimator that is unbiased for the unconditional variance. 

3.3 Bootstrap Implementation 

The object of this section is to produce a set of bootstrap replicate weights, similar to BHR or Jackknife replicate 
weights. To do this, one must make an important distinction between finite population sampling and i.i.d. sampling. 
With i.i.d. sampling the variable of interest, X , is consider random. Therefore, it’s logical to repeat the bootstrap 
sampling independently for every random variable. In finite population sampling, the response variable, X , is 
considered known for all PSUs in the frame. What is random is the sample selection variable , S j , which specifies which 

PSUs are in the sample (i.e., S j =1 means PSU j is in sample, while S . =0 means PSU j is not in sample). Since 

there is only one random variable, only one set of bootstrap samples need be generated. This can be seen in Sitter’s Rao- 
Hartley-Cochran bootstrap variance estimator. In it, the bootstrap sampling is done independent of the variable of 
interest. Therefore, once the bootstrap samples are selected, they are appropriate for any variable of interest. Likewise, 
for random sampling schemes described in the introduction. A PSU can be selected in the bootstrap selection process 
instead of a response variable without changing the procedure. Once selected, the bootstrap samples are appropriate for 
any response variable. Therefore, one set of selections can be used for all response variables. This does assume that the 
bootstrap sample size is not a function of the response variable. 

Therefore, given a bootstrap sample, an appropriate bootstrap replicate weight for PSU j for any response variable X , 

is the sum of the bootstrap weights from selected bootstrap-PSUs, which have been generated from PSU j . For the 

bootstrap procedure proposed here, it may be very difficult and time consuming to produce a set of bootstrap replicated 
/ 

weights, but that process only has to be done once. Given availability of high-speed computers, their cheap run times, 
and the usual long time period from sample selection and the production of final weights, producing a single set of 
bootstrap replicate weights is certainly practical. 

One potential difficulty with a single set of replicate weights for the proposed procedure is the process of choosing n* h 
to be more unbiased. With systematic sampling, this process is likely dependent on the response variable. Hence, each 
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variable may have a slightly different set of replicate weights. It is assumed here that one set of replicate weights that 
works well for the frame variables will also work well for other correlated variables. 

Since F hc {y ) is based on only 2 PSUs, it is not likely to be close to F hc (y) and therefore, V * (0) may not be close 

to V (0) . Hence, the bootstrap variance estimator requires the computation of a bootstrap sample size, n* h , to make 
the variance estimator more unbiased. Since there is no exact expression that relates the true systematic sampling 
variance with n h , as n h increases or decreases, determining an appropriate n h will be accomplished by a 

simulation study. To perform the simulation study frame variables are used, so estimates can be computed for any 
selected sample. The statistician always has three estimates available for this purpose. One is the measure of size or 
some function of the measure of size. The second is the estimate of the total number of PSUs (sum of the sample 
weights). The third is the average measure of size per PSU or the average per PSU of some function of the measure 
of size. If the measure of size is used in the simulation, it will be necessary to use a different year’s data to produce 
estimates; otherwise, the variances will be zero. 

To determine the appropriate n* h ' s, the simulations must first be applied to individual stratum estimates 0^ . 

Therefore, the simulation process for estimating the bootstrap variance, ^*(0^) for an estimator 0^ , works as 
follows: 

33.1 Bootstrap Procedures 

1. Select a sample ( ) from the original frame, using the methodology of the original sample design. 

2 . For the initial bootstrap sample size values, n * h , use either n h / 2 or n h . n h / 2 would be appropriate if it is 

believed that the BHR deep stratification model is appropriate. If n h is large then setting n h = n h may be 
appropriate, if one believes that the deep stratification model is an oversimplification. After the initial simulation, 
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n* h will likely require adjustment for at least some of the strata. If such is the case, it will be required to repeat the 
simulation with the new n* h ’s. 

Generate a bootstrap frame based on the selected sample. The idea behind the bootstrap frame is to use the sample 
weights ( Wj ) from the selected PSUs (j) in s . to estimate the PSU frame distribution. The bootstrap frame is 
generated in the following manner: 

For each selected PSU j , Wj bootstrap PSUs (bj ) are generated by replicating the j th PSU w j times. If w. 



has a noninteger component then a full bootstrap-PSU is generated with a reduced selection probability. The bj 



;lh 



bootstrap-PSU has the following measure of size ( m bj ): 

m bj=hj 1/v V 

where: 



( 4 ) 



1 ,= 



1 , if bj is an integer component of w j 
C, , if bj is a noninteger component of w j 
Cj being the noninteger component 



( 5 ) 



Wj : is the full - sample weight for PSU j 

Randomize the bootstrap frame according to super-population model specification. This is accomplished by placing 
the bj bootstrap-PSUs generated from PSU j within stratum h and sample in their original order of selection. 

Next, bootstrap-PSUs generated from the first PSU are paired with the next set of bootstrap-PSUs generated from 
the second PSU. The third set of bootstrap-PSUs is paired with the fourth set. This process continues until all 
bootstrap-PSUs are paired. If there are an odd number of PSUs then the last set of groupings of bootstrap-PSUs 
contains the bootstrap-PSUs generated from the last three PSUs in stratum h . This is repeated for every stratum in 
s i . Now, the bootstrap-PSUs are randomized within their respective pair. 

Select B bootstrap samples from the bootstrap frame, re-randomizing the bootstrap frame before each selection. 



The bootstrap frame, bootstrap frame ordering, measure of size ( m hJ ), and bootstrap sample size (n h ) have been 
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specified. Using these quantities select the bootstrap samples using the same procedures used to select the original 
systematic PPS sample. The one exception to this is that a bootstrap-PSU generated from noncertainy PSUs that 
become certainty in the bootstrap selection should not be eliminated from the selection process and taken in sample 
with probability 1. Their selection probability should remain unchanged and if the bootstrap-PSU is selected 
multiple times that should be reflected in the bootstrap weight (see 6 below). 

6. Compute bootstrap estimates Q* bh for each of the B bootstrap samples in an analogous manner as is done to 
compute the full sample estimate 0 |7| from S { . This is accomplished by computing a bootstrap weight, XV * , and 



then computing @* ibh the same way 0 f7| is computed, except using XV* instead of XV j . 
The bootstrap-PSU weight, XV* , is: 




Sj : is the set of all bj generated from j that are selected in the B ,h bootstrap sample, 

and 

where: 

I bj : is as previously defined 

M bj : is the number of times the bj ,h bootstrap- PSU is selected, (6) 

p b j : is the bootstrap selection probability for the bj th bootstrap-PSU. 
p bj = m bj /SI h . 



where : 

m bj : is previously defined 




7. Compute the simple variance of the Q* bh from b = 1 to B , as the Monte Carlo estimate of V* (0 l7l ) from 5, . 



O 

chic 



27 



19 



V(©„,) = 1/(S-1)£(©;„ -©;>=, where©; = 

b=\ b = 1 

8. Repeat steps 1-7, for a large number of samples, 5- , say T times. 

9. Compute the simple variance of 0 ih from i = 1 to T , V (Q h ) , as a measure of the true variance; and compute 
the average bootstrap variance V * (Q h ) , averaged over the T , V * ( Q ih ) estimates. 

V(©» ) = 1 KT - 1)2 (©» - e» )- , where 0, = 1 / 6» . 

1 = 1 » = l 

1 = 1 

10. Compare V*(0 /j )with 1^(0^) and adjust n* } to reduce the bias between V*(0^)and V(Q h ). If 
V*(Q h )is smaller than ^(0^ ) then n h should be reduced. If V^O^is larger thanV(0^) then n* h 
should be increased. Since V * (Q h ) and V (Q h ) may not be proportional to 1 !n* h , it may be difficult to predict 
how much n h should be increased or decreased. If that is the case then trial and error may be necessary. 

1 1 . Repeat steps 1-10, until the bias between V (Q h ) and V (0 /} ) has been reduced to a satisfactory level. 

12. Using the n* h from step 11, repeat steps 3-6 for the actual collected sample, generating a set of bootstrap 
replicate weights, w* that can be used to compute variances of other, more complex statistics that are not 
necessarily computed within h . 



3.4 Consistency of the Proposed Bootstrap Estimator 

To apply theorem lor 2 for the consistency of the bootstrap estimator, it must be established that F hc (y) — > F hc (y) 
and JUI CY -> JU hcY , as n -> °o . 



To do this, observe that for an arbitrary domain D : 



f 



E* 



Yj w bj X bj = Y 1 bj X bj = Y W J X j ’ since M bi has expectation p hi . 



bjeD 



bjeD 



jeD 
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where: 

£* is expectation over the bootstrap samples, 

X b j ,Xj is an arbitrary response variable defined for the bootstrap-PSUs and PSUs, respectively. 

And 

K = =1 ln'ZY- x = F„', ^E.(X- d ) = E,(Y;*) 

bjt D bje D 

where: 

y;„ 

n = 2 >* 

bjeh 

* * # * 

P h = n h ln 

I bj ,M bj and m bj are defined in (5), (4) and (6), respectively from section 3.3. 1 step 3 or step 6. 



If D t is the domain defined as the observations in a stratum partition, s !ch then 

£.(F £ ; iI ) = £.(2 ^= 1 /^ y iX 

bj€D i yeD, >eD, 

Where: 

Yjx =(T h /(p h e hj ))Xj 

T h , p h and e hJ are defined in (1), (3) and (2), respectively from section 3.1 . 

[lif y j < y 0 

Define for each j in S ich , Z jch ( y 0 ) = < . . 

0 otherwise 



and for each bj in S ich 



Y h jch (yo) — 



lif ylj^y 0 

0 otherwise 




then 
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4o-o)=a' S<) £ -( 1 'o lZ ;„ ( ,,,) = <i' E^> £ -< Kz-^yj) 

bjes icfl J bje s ich bjeD ] 

=a'5>> z * .(>■— (>»)=<«/ 

je D, jgD, jgD, jgD, 

= (! 7 S ^/ZywOW > K S fz^Oo) ) ^ 'VO'o )» aS n ch -> °° 

JED, J€D| 

where: y max is the maximum value of the y ’s in S tc/j . 

The convergence above follows from Y F f7 f v ^ converging to the population total in Z), , and 

je D, 

^y. Z / (v o ) converging to population total with y < y Q in D i . 

jeD ) 



3.5 The Variance of X * h (V* (X* t )) from the Bootstrap Frame 

In this section, an approximate expression for V* (X* h ) is derived to help explain: 1) the type of finite population 

implied by the proposed bootstrap variance estimator; and 2) how the original sample size affects the variance estimator. 
To do this, the following assumptions are made: 1) all weights are whole integers; 2) within a stratum partition the 

weights are equal and 3) n h — n h . Given these assumptions, the proposed bootstrap is closely related to the Rao, 
Hartley Cochran sampling method (Cochran 1977, pp. 266-267). The Rao, Hartley, Cochran sampling method randomly 
places the frame PSUs into n h groups. One PSU is independently selected within each group proportional to a measure 

of size z x . In the bootstrap procedures described above, within each partition, referenced by g , bootstrap-PSUs are 

randomly placed into 2 groups, referenced by n g . Given the above assumptions, the bootstrap procedures can be 

viewed as similar to as the Rao, Hartley, Cochran procedure, except that the sampling is done systematically instead of 
independently. By additionally assuming there are enough bootstrap-PSUs in g , so that the correlation of PSUs within 

g is small enough to be assumed equal to zero; and that PSUs in different g ’s have correlation » , it becomes 

possible to derive an expression for V*(X^) . The n h in p » represents the fact that the correlations are average 
(across all possible orderings) within sample correlations; as such, they will change based on the response variable, the 
original ordering and the original sample size, n h . 




22 



vn*;) = XX C0 V (x;,x;.), 

s s' 

where g and g ' represent the original frame partitioning within h, 
X* and X*. are the estimates within g and g', respectively 

= X var* (X f g ) + X X cov * <■ K > X s' )> 

8 8*8' 

= X var* (X ; ) + XX Vvar*(x;)var*(x;.) 

8 8*g' 



= X { K /(^s (^s - *)£ ^ * - Djv; (x; y + 

8 it 8 



2 ,.,-IpwM X w -« aw; <w; - w; <^; >[«;■ 2 at; <w; - daw;- <w;. - w; <*;•) 

\ leg leg ' 



(7) 



where rc* is the bootstrap sample size in g , 

N* is the number of bootstrap - PSUs in g, 

N * is the number of bootstrap - PSUs in the l' h bootstrap sampling interval, 

ties 

z hj is the m bj / X m bj = l/N r 

bjeg 

m bj is the measure of size for unit bj, 

x :=i<j=x,. 

bjeg 

The first term in (7) is the variance of the Rao, Hartley, Cochran estimator. If the P„ hgg ' J s equal zero then the 
systematic bootstrap variance is the Rao, Hartley, Cochran estimator. Since the N* ’s are equal (i.e., N* = N g / n* ) 
then the [n* /(N*(N* — 1))]V A — 1) term above equals 1 — (n* — l)/(/V* — 1) , a close approximation to 

o o o ■ ■ o o 

8 

the simple random sample finite population correction. Therefore, the proposed bootstrap variance estimator includes a 
finite population correction that reflects the extra variance reduction due to sampling without replacement. Since the 
PSUs in a sampling interval change as the sample sizes and/or ordering change, one would not expect the p n%gg » ’s, in 
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the second term, to be constant, as n h increases or decreases. They are unknown non-constant functions of n h . 

Therefore, one does not expect V* (X* ) to be proportional to 1 ln h . The l-(rt* -1)/(jV* - 1) finite population 
corrections, also makes this unlikely. Since the bootstrap variance is consistent for the original systematic variance, one 
would not expect the systematic variance, V (X h ) , to be proportional to 1 / n h , as n h increases or decreases. This is 
also demonstrated in the simulation of section 2.0. Likewise, since the bootstrap selection is systematic, one would not 
expect V" (X* h ) to be proportional to !/«,*, as n" h increases or decreases and n h fixed. 
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4.0 Simulation 

To demonstrate the advantages of the bootstrap variance estimator, a simulation study is presented comparing BHR 
and the bootstrap variance estimator. Two thousand simulations are generated using frame variables. The frame is 
the National Center for Education Statistics’ (NCES) Private School Survey (PSS). The PSS is NCES’s school frame 
for private elementary and secondary schools. Three totals (number of schools, number of teachers, and number of 
students), two averages (average students and average teachers per school), and one ratio (ratio of number of 
students to number of teachers) are estimated in the simulation. In tables 3-8, estimates are computed by each 
stratification variable (affiliation, region and school level), as well as one of the sort variables (Urbanicity). The 
School and Staffing Survey (SASS) sample design is used to select the simulation samples. Relative error, relative 
mean square error, and coverage rates are generated to evaluate the bootstrap and BHR variance estimator 
performance. 

4.1 Comparison Statistics 

In this section, the statistics used to compare the bootstrap and BHR variances are described. 

4.1.1 Relative Error 

Rel. Error = (V e (0) l/2 /V,(0) l/2 - 1) 100 

Where: V e (0) is the average of the variance estimates ( V e (0 s ) ) from either the bootstrap or BHR procedure, 

___ 2 , 00 ° 

(i.e., V e (0) is 1 / 2000 V e (O s ) , 0 S is the S rh simulation estimate of 0 ) 

s = I 

2.000 2,000 

V, (0) = 1 / 1999 £ (0 , - 0) 2 , 0 is 1 / 2000 £ 0, 

J=I J=1 



4.1.2 Relative Mean Square Error 

Rel. MSE= { [W e (0) + (V e (0) - V, (0)) 2 ]' n / V, (0)} • 100 , 



2, 000 _ 

Where: W e (&) = 1 / 1 999 £ (V e (0, ) - V, (0)) 2 . 

s = 1 



O 

ERLC 
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4.1.3 Coverage Rates 

Coverage Rate of V c (0) = 



/ 2,000 

i/20oo^y?; 



• 100 , 



Where: R‘ (0) = 



V s=l j 

1 if©, -1.96V e (©,)' /2 
0 otherwise 



<0<0, +1.96V t (©,) 1/2 



4.2 SASS Sample Design 

The sample frame, used in the simulation, is the list frame component of NCES’s Private School Survey (PSS). The list 
frame is stratified by detailed School Association (19 groups), within Association by Census Region (4 levels), and 
within Region by school level (elementary, secondary and combined). The school sample is selected using the 
systematic probability proportionate to size sampling procedure, described in the introduction. The measure of size is 
square root of the number of teachers in the school. Before sample selection, the school frame is ordered by state, school 
highest grade, urbanicity, zip code, and school enrollment. To reduce the necessary time to complete 2,000 simulation 
only one detailed school association is simulated. 

4.3 Determining n* h for the Bootstrap Variance 

As described in section 3.3, the determination of n* h requires a simulation study in itself. For each stratum, a series 
of simulations was done for various n* h . The n* h that produced the best relative error was used in the simulation 
presented below. The optimum n* h is dependent on the estimate of interest. The optimum n* h for estimating numbers 
of teachers is usually different than the optimum n h for estimating numbers of students or schools. Likewise, the 
optimum n* h for estimating averages or ratios can be different than the total optimums. Each different n* h imply a 
different set of replicate weights. Since we want only one set of replicate weights, a compromise n* h is determined 
that works reasonably well for all estimates. The results presented below use the compromise set of n* h . Table 2 
presents the values for n fl and n* h . Each simulation used in the determination of n* h had at least 250 samples. 
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Table 2.-Original ( n h ) and bootstrap ( n* h ) sample size by stratum 



Stratum 


n „ 


n l 


Stratum 


n h 


n l 


Stratum 


n h 


* 

n h 


01911 


14 


12 


01921 


10 


5 


01931 


48 


35 


01912 


16 


11 


01922 


10 


8 


01932 


46 


33 


01913 


52 


28 


01923 


10 


10 


01933 


114 


81 


01914 


34 


24 


01924 


10 


10 


01934 


52 


40 



4.4 BHR Variances 

The r th school half-sample replicate is formed using the usual textbook methodology (Wolter 1985) for establishment 
surveys with more than 2 units per stratum. This is described in the introduction. Two BHR variance estimates are 
presented. The first (BHR without FPC Adjustment) is the variance estimates described above. This estimate does not 
make any type of Finite Population Correction (FPC) adjustments. The second makes a simple FPC adjustment. The 
second BHR variance estimate (BHR with FPC Adjustment) adjusts the first variance estimator by 1 - P h , where P h is 
the average of the selection probabilities for the selected units within stratum h . 

4.5 Number of Replicates 

An important aspect of this analysis is a comparison of the stability of the two variance estimators. To do this, each 
variance estimator will have the same number of replicate estimates. Since producing bootstrap replicate weights is far 
more time consuming than producing the BHR replicate weights, it has been decided to use a relatively small number of 
replicates. Thirty-two replicates have been used in the BHR variances and thirty have been used in the bootstrap 
variances. 
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4.6 Results 

According to tables 3-8, in terms of extremes, the bootstrap variance estimator is better than either BHR variance 
estimator with respect to relative error, relative MSE, or coverage rate. The bootstrap relative errors are large in 
absolute value (greater than 20% or less than -20%) 4 times, while the BHR, with and without FPC adjustment, 
relative errors are large 17 and 12 times, respectively. 

Only 13 of the bootstrap relative MSEs are larger than 50% and only one is greater than 100%. The BHR without 
FPC adjustment has 31 relative MSEs larger than 50% and 6 greater than 100%. The FPC adjusted BHR has 26 
relative MSEs larger than 50% and 5 larger than 100%. 

The bootstrap procedure has only 2 high coverage rates (coverage rate greater than 98%) and 2 low coverage rates 
(coverage rate less than 89%). The bootstrap has 1 coverage rate greater than 99%. The BHR without FPC 
adjustment has 12 high coverage rates, 1 low coverage rate and 9 larger than 99%. Even with a FPC adjustment, the 
BHR has 10 high coverage rates, 2 low coverage rate, and 9 coverage rates greater than 99%. 

The difference between the bootstrap and BHR is largest for the Urbanicity estimates. For these estimates the BHR 
relative MSE can be almost 4 times larger than the bootstrap relative error (see tables 4 and 7 Urban). One difference 
between the Urbanicity and other estimates is the amount of sample size control in the sample design. The Urbanicity 
sample size is controlled through the sorting. Urbanicity is the third sort variable, so the control on sample size is 
small. The sample size in all other estimates is directly controlled by the stratification. One possible reason for the 
big Urbanicity differences is that the bootstrap mimics the sample process better than BHR. 

4.7 Conclusion 

This paper discussed how BHR can be used to measure the variances from surveys utilizing systematic PPS selection 
procedures. Two assumptions are necessary: 1) the extra stratification introduced by the variance stratum is sufficient 
to reflect the systematic process and 2) the variance is inversely proportional to the sample size. In table 1, it has 
been observed that systematic PPS sampling variances may not be inversely proportional to the sample size. Instead, 
a large number of times, they are a great deal more efficient and sometimes they are less efficient then the inverse 
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sample size rule would imply. One reason for this is that the clustering induced by systematic sampling changes as 
the sample size change, which makes the intercluster correlation an unknown function of the sample size. In this 
situation, the variance may not be inversely proportional to the sample size. 

To correct this problem, a bootstrap variance estimator has been introduced which does not make the inverse sample 
size assumption. Given an appropriate super-population model, the bootstrap procedure produces consistent variance 
estimates. It has also been demonstrated that the bootstrap procedure adjusts for without replacement sampling. 
Based on the simulation of the SASS survey design (tables 3-8), the bootstrap variance estimator performs better the 
BHR with respect to relative error, relative MSE and coverage rates. This is especially true with the Urbanicity 
estimates. This remains true even after a simple finite population adjustment is made to the BHR. One drawback of 
the proposed bootstrap procedure is that the determination of an appropriate bootstrap sample size can only be 
implemented using frame variables. However, with appropriate frame variables, the bootstrap variances are close to 
unbiased, even when the super-population model assumption fails. 
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Table 3. — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for students estimates by affiliation, 
region, level and urbanicity 





Bootstrap 


BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-0.3 


30.0 


93.2 


12.4 


44.1 


96.1 


9.0 


35.3 


96.1 


Northeast 


-10.7 


43.7 


90.4 


1.9 


47.7 


94.9 


-0.7 


45.4 


94.8 


Midwest 


-1.9 


46.0 


93.4 


11.2 


45.6 


97.1 


8.1 


40.7 


97.0 


South 


15.7 


55.8 


96.2 


25.4 


66.4 


99.8 


20.6 


55.0 


99.7 


West 


-2.0 


38.6 


92.1 


12.8 


47.9 


94.7 


9.9 


42.8 


94.6 


Elementary 


-12.8 


38.0 


89.9 


-6.3 


28.7 


93.1 


-8.3 


29.6 


91.9 


Secondary 


-7.5 


48.7 


88.6 


5.7 


45.2 


94.6 


-6.1 


34.9 


.94.2 


Combined 


13.3 


51.2 


96.3 


26.0 


71.2 


96.1 


21.9 


61.6 


96.0 


Rural 


16.6 


54.4 


97.1 


32.8 


89.1 


97.3 


28.6 


78.6 


97.1 


Suburban 


8.0 


38.9 


94.9 


-0.9 


26.5 


92.4 


-3.8 


26.0 


92.4 


Urban 


31.6 


90.4 


98.2 


48.0 


132.1 


100.0 


43.7 


119.4 


100.0 



Table 4. — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for schools estimates by affiliation, 
region, level and urbanicity 





Bootstrap 


BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-4.5 


27.8 


93.1 


12.7 


40.7 


97.0 


9.7 


35.4 


97.0 


Northeast 


4.3 


43.6 


94.6 


10.3 


52.3 


94.9 


8.0 


49.0 


94.9 


Midwest 


4.2 


42.8 


92.9 


12.5 


51.4 


98.3 


9.8 


46.9 


95.7 


South 


-10.9 


32.7 


90.7 


-6.6 


26.5 


89.6 


-10.3 


29.0 


89.4 


West 


-2.4 


35.1 


92.9 


7.8 


43.7 


92.2 


5.1 


40.0 


92.2 


Elementary 


1.3 


34.9 


93.6 


16.1 


57.0 


95.9 


14.0 


52.9 


95.9 


Secondary 


-2.9 


57.0 


90.5 


26.3 


107.1 


97.2 


14.8 


81.9 


95.9 


Combined 


-6.2 


29.5 


91.2 


-1.1 


28.2 


92.3 


-4.2 


27.7 


92.3 


Rural 


7.5 


36.8 


95.7 


24.2 


71.2 


98.7 


20.9 


63.8 


98.7 


Suburban 


6.5 


36.6 


95.0 


23.1 


67.5 


97.4 


19.9 


60.6 


97.4 


Urban 


11.5 


43.2 


96.1 


53.7 


147.6 


97.5 


49.5 


135.0 


97.5 
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Table 5 . — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for teachers estimates by affiliation, 
region, level and urbanicity 





Bootstrap 




BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-6.6 


28.8 


92.4 


-4.5 


25.5 


92.3 


-7.5 


26.6 


91.0 


Northeast 


-4.1 


34.5 


92.5 


-8.6 


33.3 


94.4 


-11.0 


34.5 


94.3 


Midwest 


-6.2 


32.6 


90.3 


15.5 


49.6 


98.4 


12.4 


f 43.5 


97.1 


South 


-10.5 


30.7 


89.4 


-11.3 


27.7 


83.7 


-14.9 


31.9 


83.5 


West 


9.6 


52.1 


96.1 


16.4 


60.7 


96.4 


13.3 


54.2 


95.1 


Elementary 


3.9 


37.7 


93.6 


18.1 


63.2 


97.1 


15.6 


58.3 


97.1 


Secondary 


17.8 


75.5 


95.4 


31.0 


98.3 


97.5 


17.0 


64.9 


93.7 


Combined 


-6.5 


29.7 


91.9 


-7.3 


27.6 


89.9 


-10.3 


29.6 


89.9 


Rural 


20.3 


60.5 


97.5 


29.5 


79.3 


97.4 


25.4 


69.0 


97.3 


Suburban 


11.1 


41.7 


96.1 


4.8 


30.7 


92.4 


1.6 


27.4 


92.4 


Urban 


49.7 


139.8 


99.3 


81.7 


244.0 


100.0 


76.0 


222.9 


100.0 



Table 6 . — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for students per school estimates by 
affiliation, region, level and urbanicity 





Bootstrap 




BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-2.9 


28.3 


92.7 


11.7 


41.2 


97.4 


8.5 


35.7 


96.2 


Northeast 


-2.6 


39.7 


93.0 


2.6 


42.8 


93.6 


0.1 


40.5 


92.4 


Midwest 


1.6 


39.4 


94.0 


19.5 


63.5 


96.0 


16.2 


56.4 


96.0 


South 


1.6 


32.7 


93.6 


9.4 


32.4 


96.0 


5.1 


25.8 


94.5 


West 


-2.1 


37.9 


92.8 


9.2 


39.5 


96.0 


6.4 


35.2 


95.9 


Elementary 


-8.7 


33.5 


91.0 


-4.3 


27.1 


94.5 


-6.2 


27.5 


94.4 


Secondary 


1.0 


53.0 


92.2 


13.2 


60.7 


94.8 


1.2 


41.6 


93.4 


Combined 


5.2 


36.7 


95.1 


17.4 


52.2 


95.0 


13.6 


44.4 


94.9 


Rural 


5.8 


41.4 


94.6 


45.7 


131.2 


99.6 


41.3 


118.2 


99.6 


Suburban 


-12.9 


35.7 


89.1 


-8.3 


32.7 


92.1 


-10.9 


33.9 


90.8 


Urban 


10.3 


44.4 


95.9 


32.1 


91.7 


97.3 


28.2 


81.6 


97.2 
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Table 7 . — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for teachers per school estimates by 
affiliation, region, level and urbanicity 





Bootstrap 




BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-5.8 


28.4 


92.4 


4.0 


27.3 


95.9 


1.0 


24.6 


95.8 


Northeast 


2.1 


42.1 


93.7 


0.6 


41.2 


90.9 


-1.7 


39.4 


90.8 


Midwest 


-0.7 


37.7 


92.5 


18.2 


60.7 


99.7 


15.0 


54.0 


99.7 


South 


-10.9 


32.5 


89.4 


-9.6 


28.0 


89.4 


-13.2 


31.4 


88.1 


West 


5.5 


41.3 


95.1 


12.1 


45.4 


93.6 


9.2 


40.1 


93.6 


Elementary 


4.6 


38.7 


94.0 


17.7 


57.2 


97.1 


15.3 


52.4 


97.1 


Secondary 


8.6 


54.2 


95.2 


29.4 


93.3 


97.4 


16.6 


63.8 


93.7 


Combined 


-6.9 


29.9 


91.6 


-4.1 


26.2 


92.3 


-7.2 


27.1 


91.0 


Rural 


1.1 


37.4 


93.2 


27.9 


83.6 


99.6 


24.2 


74.6 


99.6 


Suburban 


-10.7 


34.4 


89.7 


-2.9 


34.9 


91.8 


-5.5 


34.3 


91.8 


Urban 


10.6 


44.9 


95.5 


61.6 


177.4 


99.8 


56.7 


161.4 


99.8 



Table 8 . — % relative error, % relative mean square error and % coverage rates for the 
bootstrap and BHR variance estimator for students/teacher ratio estimates by 
affiliation, region, level and urbanicity 





Bootstrap 




BHR without FPC 


BHR with FPC 










Adjustment 


Adjustment 




Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Rel. 


Rel. 


Cov. 


Estimate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Error 


MSE 


Rate 


Other Affil. 


-0.3 


31.9 


94.2 


12.3 


46.0 


96.1 


9.1 


40.6 


94.7 


Northeast 


-4.3 


56.3 


91.6 


9.3 


70.4 


94.7 


6.7 


66.4 


94.7 


Midwest 


-5.5 


67.1 


91.2 


3.1 


43.9 


95.4 


0.5 


41.6 


94.1 


South 


6.7 


45.2 


95.4 


3.2 


31.5 


95.9 


-0.8 


28.6 


93.4 


West 


-1.0 


38.3 


93.9 


9.8 


46.9 


97.3 


7.1 


42.8 


97.3 


Elementary 


-2.5 


43.9 


93.2 


11.8 


52.2 


99.5 


9.5 


48.5 


99.5 


Secondary 


-25.3 


49.1 


81.1 


1.2 


33.2 


94.1 


-9.3 


32.5 


91.4 


Combined 


9.7 


46.2 


95.8 


16.3 


53.0 


95.9 


12.6 


45.7 


95.7 


Rural 


7.0 


59.8 


95.3 


22.7 


84.2 


99.9 


18.9 


75.6 


99.8 


Suburban 


1.1 


37.6 


93.5 


18.0 


61.6 


97.1 


14.8 


55.3 


97.1 


Urban 


5.4 


45.6 


94.7 


15.0 


58.3 


93.8 


11.6 


52.1 


93.7 
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Appendix 

In this section, the consistency of the proposed bootstrap estimators is proven. 

Theorem 

Given the superpopulation model described in section 3.1, the conditional bootstrap variance estimator of 
*-22 W h jXj , given the model and the sample, is consistent, as n — » °° , provided K(y)-> F kc (y) and 

h jeh 

Myhc n-^oo. 



Before the conditional result is proven, additional terms, definitions and results must be stated. 
Definitions 



Mallows’ distance 
Let 

3^ = {all distributions on 91 s } 

= :jH r ^(x)<oo} 

For two distributions H and G in 3 r s , their Mallow’s distance is 
p r (H,G) = inf(E\\X-Y\\ r ) Ur , 

r X.Y 

where r x Y is the collection of all possible joint distributions of the pairs (X, Y) whose marginal 
distributions are H and G, respectively. For random U and V having distributions He 3 rJ 
andGe 3 ^respectively, define p r (f/, V) = p r (H,G). 



Let X, X n be observations from a stochastic model P and 91 n = 91 n (X X n ,P n ) be a random variable. 
The sampling distribution of 91 n is given by: 
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The bootstrap sampling distribution of 9? n is given by: 

H boot (*^) = P* * n w here P n is an estimator of /phased on X { ,...,X n and P [ represents the 

probability with respect to P n . 9?* = (X*,..., X* n ), where {X* V . M X* ) is a bootstrap sample from P n . 

Define = ^q ch M Y ch > where <lch = n ch 1 n h and /*?=/** = ' The bootstra P mean ( u }„h and 

ce/i h 

U*~) are defined in an analogous manner where P n is used instead of P n . jLl*-^ h = ^ J <J C f } My c h = ^/i anc * 



ce/i 

^y=K =X^>V.=5>a^- 



h 



h 



Results (all from Bickel and Freedman 1981) 



Let U and V be random vectors and a be a constant. Then, 
p r (aU,aV) = \a\p r (U,V) A.l 



If £||f/|| < 00 and £|V|| < 00 , then 

[p 2 (U,V)f =[p 2 (U-EU,V-EV)f +\\EU-EVf A.l 



Let w and W be two sequences of independent random vectors whose distributions are in 3 rj and 
EU : = EV, for all j . Then, 

J J J 



Pi 



m m 

2X-2>; 



>= I 



J- 



s V,jf 



A.3 



j = 1 
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In the discussion that follows, we will always assume that the partitioning is known and that the selected sample ( s { ) 
is given. 

Given the above assumptions, definitions and results, we will investigate the properties of the bootstrap distribution 
of the total X = Y Y w hi X . (i.e., H B00T (x) , where 9T = yfn(X* -X)), as n — It will be shown, 

h jeh 

using Mallow’s distance ( ), that the bootstrap distribution is consistent. It then follows that the bootstrap variance 

is consistent (Shao and Tu 1995). The proof closely follows example 3.1 in (Shao and Tu 1995), which was also 
used by Bickel and Freedman (1981). 

There are many ways of generating a bootstrap sample from Tp„u.r=u Y f * be one such bootstrap, which 

* * 

also defines a bootstrap for X . The important thing is that F hc (y) — > F hc ( y ) and JU hcY — > fl hcY , as n — » . 

Given a method of generating a bootstrap sample, X •=ix W h jXj can be computed. 

h my; 
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p 2 (. H boot ,H n ) = p 2 (^(X , -X),^(X-jU i )) 



^Z p k (y A * - y h ) j, X Ph ( y h -My h „ ) j 

^ -y*))Vn(p A (y A -/^)J , [by A.3] 



BOOT'H n ) — ^/?2 



/ — (y* ^/i)^ / — (3^A /^v fc /i )) 

Vn vn 



=- Xa 



J=l 



«/, ( »h 

^( y ] h - y h ) . 



-|2 



J=I 



yj 



- - Z Z Z Pi ( 0 * “ y* )• (y * - /*>v* )) 

n h c=I j=I 

1 C/i n, ‘ / 

= “ Z Z Z P 2 (( yi*c - 5v ). ( yi*f - Myhc )) [by exchangeability] 

n h c=\ j = 1 

fit 



C |, «, 



[by A.l] 
[by A.3] 



1 / \2 
= -ZZ n cP 2 ((y.*/,c -y* f ).(yi*c -M**)) 

n h c = 1 

=-Z^ n cfe(yr*c*yi*cT -|| £ yi/t -^.y^rl [by a.2] 

n /, r= i L J 

= Z Z I* \p2 (yL > y\hc 1 - I^iac - E . >w II" 1 

h C=l L J 

= Z Z Vch \p 2 {Kc ’ F hcJ - 1 Mhcr - Myhc I" 1 [by definition] 

h c= 1 L -I 

—> 1) —> o( 1) [by condition of theorem] 



This establishes the consistency of both H B 00 T (x) and the bootstrap variance for X given the partitioning and the 
sample s f and a realization of for all h on Sj . The consistency was established without knowledge of the 
distribution function F . 
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Careers: Critical Features of a Truly Longitudinal 
Study 


Dan Kasprzyk 
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96-03 (Feb.) 


National Education Longitudinal Study of 1988 
(NELS:88) Research Framework and Issues 


Jeffrey Owings 
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Census Mapping Project/School District Data Book 


Tai Phan 


96-05 (Feb.) 


Cognitive Research on the Teacher Listing Form for 
the Schools and Staffing Survey 


Dan Kasprzyk 


96-06 (Mar.) 


The Schools and Staffing Survey (SASS) for 1998-99: 
Design Recommendations to Inform Broad Education 
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Dan Kasprzyk 


96-07 (Mar.) 
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Teacher Effectiveness? 


Dan Kasprzyk 


96-08 (Apr.) 


How Accurate are Teacher Judgments of Students’ 
Academic Performance? 
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Dan Kasprzyk 


96-10 (Apr.) 
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Towards an Organizational Database on America’s 
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The 1995 National Household Education Survey: 
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Schools 
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Andrew G. 
Malizio 


96-18 (Aug.) 
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Behaviors, and Approaches to Learning with Young 
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96-19 (Oct.) 


Assessment and Analysis of School-Level 
Expenditures 
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96-20 (Oct.) 


1991 National Household Education Survey 
(NHES:91) Questionnaires: Screener, Early 
Childhood Education, and Adult Education 


Kathryn Chandler 


96-21 (Oct.) 
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(NHES:93) Questionnaires: Screener, School 
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1995 National Household Education Survey 
(NHES:95) Questionnaires: Screener, Early 
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96-23 (Oct.) 
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Dan Kasprzyk 


96-25 (Oct.) 
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Suggested Items for the 1998-1999 Schools and 
Staffing Survey 
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Surveys for 1993-94 
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1991 and 1995 National Household Education Survey 
Questionnaires: NHES:91 Screener, NHES:91 Adult 
Education, NHES:95 Basic Screener, and NHES:95 
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Procedures in the 1993 National Household Education 
Survey (NHES:93) 

Unit and Item Response, Weighting, and Imputation 
Procedures in the 1995 National Household Education 
Survey (NHES:95) 

The Determinants of Per-Pupil Expenditures in 
Private Elementary and Secondary Schools: An 
Exploratory Analysis 

Design, Data Collection, Interview Timing, and Data 
Editing in the 1995 National Household Education 
Survey 
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Private School Teacher Questionnaires for the Schools 
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Dan Kasprzyk 


97-11 (Apr.) 


International Comparisons of Inservice Professional 
Development 


Dan Kasprzyk 


97-12 (Apr.) 


Measuring School Reform: Recommendations for 
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97-13 (Apr.) 


Improving Data Quality in NCES: Database-to-Report 
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Susan Ahmed 


97-14 (Apr.) 


Optimal Choice of Periodicities for the Schools and 
Staffing Survey: Modeling and Analysis 


Steven Kaufman 


97-15 (May) 


Customer Service Survey: Common Core of Data 
Coordinators 


Lee Hoffman 


97-16 (May) 


International Education Expenditure Comparability 
Study: Final Report, Volume I 


Shelley Bums 


97-17 (May) 


International Education Expenditure Comparability 
Study: Final Report, Volume n, Quantitative Analysis 
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Shelley Bums 


97-18 (June) 


Improving the Mail Return Rates of SASS Surveys: A 
Review of the Literature 


Steven Kaufman 


97-19 (June) 


National Household Education Survey of 1995: Adult 
Education Course Coding Manual 


Peter Stowe 


97-20 (June) 


National Household Education Survey of 1995: Adult 
Education Course Code Merge Files User’s Guide 


Peter Stowe 


97-21 (June) 


Statistics for Policymakers or Everything You Wanted 
to Know About Statistics But Thought You Could 
Never Understand 


Susan Ahmed 


97-22 (July) 


Collection of Private School Finance Data: 
Development of a Questionnaire 


Stephen 

Broughman 
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97-23 (July) 


Further Cognitive Research on the Schools and 
Staffing Survey (SASS) Teacher Listing Form 


Dan Kasprzyk 


97-24 (Aug.) 


Formulating a Design for the ECLS: A Review of 
Longitudinal Studies 


Jerry West 


97-25 (Aug.) 


1996 National Household Education Survey 
(NHES.96) Questionnaires: Screener/Household and 
Library, Parent and Family Involvement in Education 
and Civic Involvement, Youth Civic Involvement, and 
Adult Civic Involvement 


Kathryn Chandler 


97-26 (Oct.) 


Strategies for Improving Accuracy of Postsecondary 
Faculty Lists 


Linda Zimbler 


97-27 (Oct.) 


Pilot Test of IPEDS Finance Survey 


Peter Stowe 


97-28 (Oct.) 


Comparison of Estimates in the 1996 National 
Household Education Survey 
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97-29 (Oct.) 


Can State Assessment Data be Used to Reduce State 
NAEP Sample Sizes? 


Steven Gorman 


97-30 (Oct.) 


ACT’s NAEP Redesign Project: Assessment Design is 
the Key to Useful and Stable Assessment Results 


Steven Gorman 


97-31 (Oct.) 


NAEP Reconfigured: An Integrated Redesign of the 
National Assessment of Educational Progress 


Steven Gorman 


97-32 (Oct.) 


Innovative Solutions to Intractable Large Scale 
Assessment (Problem 2: Background Questionnaires) 


Steven Gorman 


97-33 (Oct.) 


Adult Literacy: An International Perspective 


Marilyn Binkley 


97-34 (Oct.) 


Comparison of Estimates from the 1993 National 
Household Education Survey 


Kathryn Chandler 


97-35 (Oct.) 


Design, Data Collection, Interview Administration 
Time, and Data Editing in the 1996 National 
Household Education Survey 
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97-36 (Oct.) 


Measuring the Quality of Program Environments in 
Head Start and Other Early Childhood Programs: A 
Review and Recommendations for Future Research 


Jerry West 
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NAEP Open-ended Items 
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97-38 (Nov.) 


Reinterview Results for the Parent and Youth 
Components of the 1996 National Household 
Education Survey 


Kathryn Chandler 


97-39 (Nov.) 


Undercoverage Bias in Estimates of Characteristics of 
Households and Adults in the 1996 National 
Household Education Survey 
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Unit and Item Response Rates, Weighting, and 
Imputation Procedures in the 1996 National 
Household Education Survey 
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97-41 (Dec.) 


Selected Papers on the Schools and Staffing Survey: 
Papers Presented at the 1997 Meeting of the American 
Statistical Association 


Steve Kaufman 
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Improving the Measurement of Staffing Resources at 
the School Level: The Development of 
Recommendations for NCES for the Schools and 
Staffing Survey (SASS) 


Mary Rollefson 


97-43 (Dec.) 


Measuring Inflation in Public School Costs 


William J. Fowler, 
Jr. 


97-44 (Dec.) 


Development of a SASS 1993-94 School-Level 
Student Achievement Subfile: Using State 
Assessments and State NAEP, Feasibility Study 


Michael Ross 


98-01 (Jan.) 


Collection of Public School Expenditure Data: 
Development of a Questionnaire 


Stephen 

Broughman 


98-02 (Jan.) 


Response Variance in the 1993-94 Schools and 
Staffing Survey: A Reinterview Report 


Steven Kaufman 


98-03 (Feb.) 


Adult Education in the 1990s: A Report on the 1991 
National Household Education Survey 


Peter Stowe 


98-04 (Feb.) 


Geographic Variations in Public Schools’ Costs 


William J. Fowler, 



Jr. 
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The Redesign of the Schools and Staffing Survey for 
1999-2000: A Position Paper 
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Adult Education Participation Decisions and Barriers: 
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Sampling 



Contact 

Steven Kaufman 

Ralph Lee 

Tai Phan 
Dan Kasprzyk 
Jeffrey Owings 

Peter Stowe 

Aurora D’Amico 
Steven Kaufman 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 







TM030095 



NOTICE 

REPRODUCTION BASIS 




This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




EFF-089 (9/97) 




